Enhanced 3d interfacing for remote devices

ABSTRACT

Operating a computerized system includes presenting user interface elements on a display screen. A first gesture made in a three-dimensional space by a part of a body of a user is detected. In response to the first gesture, an area of the display screen selected by the user is identified, and a magnification level of one or more of the user elements appearing in the selected area on the display screen is increased. After increasing the magnification level, a second gesture made by the part of the body of the user is detected so as to select one of the user interface elements that appear in the selected area.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/721,582, filed Mar. 11, 2010, which claims the benefit of U.S.Provisional Application No. 61/159,808 filed Mar. 13, 2009. Both ofthese related applications are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates generally to user interfaces for computerizedsystems and specifically to user interfaces with three-dimensionalcharacteristics.

2. Description of the Related Art

Many different types of user interface devices and methods are currentlyavailable. Common tactile interface devices include the computerkeyboard, mouse and joystick. Touch screens detect the presence andlocation of a touch by a finger or other object within the display area.Infrared remote controls are widely used, and “wearable” hardwaredevices have been developed, as well, for purposes of remote control.

Computer interfaces based on three-dimensional sensing of parts of theuser's body have also been proposed. For example, PCT InternationalPublication WO 03/071410, whose disclosure is incorporated herein byreference, describes a gesture recognition system using depth-perceptivesensors. A three-dimensional sensor provides position information, whichis used to identify gestures created by a body part of interest. Thegestures are recognized based on the shape of the body part and itsposition and orientation over an interval. The gesture is classified fordetermining an input into a related electronic device.

As another example, U.S. Pat. No. 7,348,963, whose disclosure isincorporated herein by reference, describes an interactive video displaysystem, in which a display screen displays a visual image, and a cameracaptures three-dimensional information regarding an object in aninteractive area located in front of the display screen. A computersystem directs the display screen to change the visual image in responseto the object.

SUMMARY

An embodiment of the invention provides a method for operating acomputerized system, which is carried out by presenting user interfaceelements on a display screen of the computerized system and detecting afirst gesture made in a three-dimensional space by a distal portion ofan upper extremity of a user while a segment of the distal portionthereof rests on a surface. In response to the first gesture, an area ofthe display screen selected by the user is identified, and acorresponding user interface element is displayed. After displaying thecorresponding user interface element, a second gesture made by thedistal portion while the segment continues to rest on the surface isdetected so as to select one of the user interface elements that appearsin the selected area.

In another embodiment, the method further includes mapping an operationto the corresponding user interface element, wherein the second gesturecauses the operation to be performed.

In yet another embodiment, the method further includes mapping athree-dimensional location of the distal portion of the upper extremityto two parameters of a two-dimensional parametric surface, which is asection of a sphere that corresponds to possible locations of the distalportion of the upper extremity while the segment of the distal portionrests on a surface, and mapping the two parameters to correspondingparameters in a planar two-dimensional coordinate system of the displayscreen where the user interface elements are located

According to an aspect of the method, the first gesture describes afirst arc-like movement forming at least a portion of a horizontalcircular arc mapped to the section of the sphere as a pointing command,and the second gesture includes a second arc-like movement mapped to thesection of the sphere as a selection command.

According to an additional aspect of the method, the pointing commandincludes pointing to a letter on the display screen and the selectioncommand includes inputting the letter to the system.

According to one aspect of the method, a threshold is defined, andincludes displaying a subset of letters when a magnitude of the arc-likemovement is less than the threshold and inputting the letter when themagnitude of the arc-like movement is greater than the threshold.

According to an aspect of the method, the subset of letters is shiftedon the display screen using a language model to determine a probabilityof a preferred letter, and wherein inputting the letter is performed ina single continuous motion with high probability.

According to one aspect of the method, the segment of the distal portionmay include an elbow, a wrist, or a forearm.

An embodiment of the invention provides a method for operating acomputerized system, which is carried out by presenting user interfaceelements on a display screen of the computerized system and detecting afirst gesture made in a three-dimensional space by a part of a body of auser. An area of the display screen selected by the user is identifiedresponsively to the first gesture, and a magnification level of one ormore of the user elements appearing in the selected area on the displayscreen is increased. After increasing the magnification level, a secondgesture made by the part of the body of the user is detected so as toselect one of the user elements that appear in the selected area.

According to an aspect of the method, a third gesture made by the partof the body is detected, and the magnification level is decreased inresponse to the third gesture.

According to an additional aspect of the method, the first and the thirdgesture include circular motions of a hand of the user in opposite,respective directions.

According to one aspect of the method, detecting the second gestureincludes actuating a shortcut on the display in response to the secondgesture.

According to yet another aspect of the method, selecting the magnifiedalphanumeric symbol includes adding the selected magnified alphanumericsymbol to a word spelled on the display screen, wherein the methodincludes detecting a third gesture made by the part of the body,opposite to the second gesture, and removing one or more symbols fromthe word in response to the third gesture.

According to one aspect of the method, the first gesture includes athree-dimensional movement by the part of the body of the user.

According to still another aspect of the method, presenting userinterface elements includes displaying a plurality of symbols arrangedin at least one arc.

According to a further aspect of the method, displaying the plurality ofsymbols includes presenting a set of symbols in a first arc, andincreasing the magnification level includes presenting a magnifiedsubset of the set of symbols in a second arc adjacent to the first arc.

According to an aspect of the method, detecting the first gestureincludes detecting an arcuate movement of a hand of the user, andassociating the arcuate movement with the plurality of symbols in the atleast one arc.

According to an additional aspect of the method, presenting userinterface elements includes presenting a sequence of textual characters,and increasing the magnification level includes displaying furthercharacters for addition to the sequence using a language model to selectthe further characters.

According to another aspect of the method, detecting the first gestureincludes scrolling forward or backward along the sequence responsivelyto first movements of a hand of the user in first and second directionsalong the sequence, and detecting the second gesture includes selectingthe further characters for addition to the sequence in response tosecond movements of the hand in at least a third direction perpendicularto the first and second directions.

An embodiment of the invention provides a computer software product foroperating a computer system, including a sensing device, which isconfigured to detect at least a part of a body of a user, a displayscreen, which is configured to present user interface elements, and aprocessor, which is coupled to the sensing device so as to detect afirst gesture made in a three-dimensional space by the part of the body.The processor is additionally configured to identify an area of thedisplay screen selected by the user in response to the first gesture,and to increase a magnification level of one or more of the userinterface elements appearing in the selected area on the display screen,and after increasing the magnification level, to detect a second gesturemade by the part of the body so as to select one of the user interfaceelements that appears in the selected area.

An embodiment of the invention provides a computer software product foroperating a computerized system, including a computer storage medium inwhich computer program instructions are stored, which instructions, whenexecuted by a computer, cause the computer to present user interfaceelements on a display screen of the computerized system, to detect afirst gesture made in a three-dimensional space by a part of a body of auser, to identify an area of the display screen selected by the user inresponse to the first gesture, and to increase a magnification level ofone or more of the user interface elements appearing in the selectedarea on the display screen. After increasing the magnification level,the instructions cause the computer to detect a second gesture made bythe part of the body of the user so as to select one of the userinterface elements that appears in the selected area.

There is also provided, in accordance with an embodiment of the presentinvention, a method for operating a computerized system, including thesteps of presenting user interface elements on a display screen of thecomputerized system and detecting a gesture made in a three-dimensionalspace by a part of a body of a user. While the user performs thegesture, one or more of the user interface elements on the displayscreen are continuously modified responsively to a direction of thegesture.

In some embodiments, continuously modifying the one or more of the userinterface elements includes increasing or decreasing a magnificationlevel of at least one of the user interface elements, typically byzooming in on a user interface element toward which the gesture isdirected.

In other embodiments, presenting the user interface elements includespresenting a sequence of textual characters, which is continuouslymodifying by adding characters to the sequence while scrolling over thesequence responsively to the gesture. Adding the characters typicallyincludes presenting choices of further characters to add to thesequence, using a language model to determine the choices, and selectingat least one of the choices responsively to the gesture. Presenting thechoices may include determining, based on the language model, arespective likelihood of correctness of each of the choices, anddisplaying the choices so that an effort required by the user to selecta given choice is a decreasing function of the likelihood. Additionallyor alternatively, continuously modifying the one or more of the userinterface elements may include scrolling forward or backward along thesequence responsively to first movements of a hand of the user in firstand second directions along the sequence, and selecting furthercharacters for addition to the sequence responsively to second movementsof the hand in at least a third direction perpendicular to the first andsecond directions.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

For a better understanding of the present invention, reference is madeto the detailed description of the invention, by way of example, whichis to be read in conjunction with the following drawings, wherein likeelements are given like reference numerals, and wherein:

FIG. 1 is a schematic, pictorial illustration of a system for remotegesture-mediated information input, in accordance with an embodiment ofthe present invention;

FIG. 2 is a view of portions of a system operating under remote controlof a user in accordance with an embodiment of the present invention;

FIG. 3 is a view of portions of a system operating under remote controlof a user in accordance with an embodiment of the present invention;

FIG. 4 is a schematic, pictorial illustration of a system for remoteinformation input, in accordance with an embodiment of the presentinvention;

FIG. 5 is a view of portions of a system operating under remote controlof a user in accordance with an embodiment of the present invention;

FIG. 6 is a view of portions of a system operating under remote controlof a user in accordance with an embodiment of the present invention;

FIG. 7 is a view of portions of a system operating under remote controlof a user in accordance with an embodiment of the present invention; and

FIG. 8 is a flow chart of a method for remotely interfacing with acomputer system, in accordance with an embodiment of the presentinvention.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth inorder to provide a thorough understanding of the various principles ofthe present invention. It will be apparent to one skilled in the art,however, that not all these details are necessarily always needed forpracticing the present invention. In this instance, well-known circuits,control logic, and the details of computer program instructions forconventional algorithms and processes have not been shown in detail inorder not to obscure the general concepts unnecessarily.

Overview.

Embodiments of the present invention that are described hereinbelowprovide improved methods, products and systems for providing remoteinput to an electronic device. When a user interacts remotely with adevice that requires accurate input and navigation decisions, theseembodiments provide an intuitive and streamlined interface.

Remote input may be provided for interaction with a remote device suchas a gaming console, an interactive television, a computerized cellularphone, or a computer. In the context of the present application andclaims, the term “remote device” herein refers to any remotelygovernable device containing a processing unit. A sensing device may beused to detect a virtual control, such as a virtual keyboard. Thesensing device is typically a three-dimensional camera that detectsinformation that includes the position of a body (or at least parts ofthe body) of the user or other tangible entities wielded or operated bythe user for interacting with a computer application running on theremote device, all of which are sometimes referred to herein forconvenience as “control entities”. The sensing device detects thepresence and changes of position of a control entity, i.e. its speed anddirection. The remote device interprets movements detected by thesensing device as described hereinbelow.

The virtual control is positioned in front of the user, usually betweenthe user and a display. The remote device presents user interfaceelements on the display. The sensing device detects the movements of thecontrol entity in a three-dimensional space, such as a user's handmanipulating the virtual control, and translates them into commands forthe remote device. For example, movement of the control entity using acircular gesture may be interpreted by the remote device as a command toadjust a magnification (or zoom) level of a remote information inputinterface comprising the user interface elements on the display.“Magnification” in this context is not limited to simple visualmagnification: New details or relevant potentially selected options maybe exposed. In movements referred to herein as “3-dimensional”movements, a distance between the control entity and the sensing devicevaries as a gesture, e.g., the circular gesture, is performed. Movementsin which the distance between the control entity and the sensing deviceremains substantially constant are classified as either “1-dimensional”or “2-dimensional movements”. Performing a clockwise circular gesturewith the control entity may command the remote device to increase themagnification level of the display, whereas a counter-clockwise circularmovement may cause minification. The terms “clockwise” and“counterclockwise” are used arbitrarily herein to distinguish twomeaningful gestures. These terms have no necessary physical meaningswith respect to the actual configuration of the embodiments.

Thus, by interpretation of the user's gestures, the remote device causesthe interface to zoom in and out, with a potentially infinite zoomrange. The actual distance between the user and the remote device,however, remains largely constant. Conceptually, the result of thisarrangement is that the interface is essentially three-dimensional, andthe elements of the interface may be continuously regrouped inthree-dimensional space so as to move toward or away from the user asrequired, depending on the direction of the user's gesture. Thus, forexample, the display may zoom in on a user interface element towardwhich a gesture is directed. This zoom may continue until the user'shand reaches the virtual location of the user element in thethree-dimensional space, whereupon the element is selected (with orwithout an additional selection gesture).

In other embodiments of the invention, the virtual control may be avirtual keyboard for remote information input, such as remote textinput. The display presents a remote information input interfacerepresenting the virtual keyboard. The symbol positions, e.g., therelative positions of the letters in the virtual keyboard, may be heldconstant. The relative size of the symbols may be varied on the displayin response to movements by the control entity. The distance of thecontrol entity from the sensing unit and the relative motion of thecontrol entity with respect to the sensing unit may be interpreted as amanipulation of the virtual control, and hence as commands to the remotedevice as described hereinbelow. The virtual control may be interpretedby the remote device to be constructed or oriented so as to accommodateright or left handed operation as the case may be. The positions of thesymbols in the virtual keyboard may be held constant, while the relativesize of the symbols may be varied to reflect expected relevance orlikelihood of each symbol being a suggested next symbol for remoteinput, e.g., based upon context awareness.

Movement of the control entity forward towards the sensing unit maycause a selected symbol, e.g., a letter to be input. Movement of thecontrol entity away from the sensing unit may cause the previously inputsymbol to be erased. Selection is typically performed in two stages. Afirst selection stage may be used to determine a specific group ofsymbols. A second selection stage is used to input an individual symbolfrom the specific group of symbols. In some embodiments, the firstselection stage is made by an arc-like movement of the control entity,e.g., clockwise or counter-clockwise, followed by a slight movementtowards the sensing unit. A threshold, which may be fixed or adaptive,is defined to enable the remote device to detect completion of eachselection stage. The remote device interprets a movement with amagnitude less than the threshold as completion of the first selectionstage. When the magnitude of the movement is greater than the threshold,the remote device interprets the movement as completion of the secondstage. The values of the threshold may depend upon the resolvingcapabilities of the sensing device, and are typically set so as todetect significant motion, while ignoring “jitter” by the controlentity. In some cases, there is no pause between the stages, e.g., whena single continuous movement is performed. The second selection stagemay constitute an additional arc-like movement by the control entitytowards the sensing unit and downwards, as described in more detailhereinbelow.

Selection stages may be combinations of continuous and discrete, curvedand linear movements in many directions with respect to the sensingunit. In any case the user can execute the combinations while restinghis elbow (or other relatively proximal segment of his upper extremity,e.g., a wrist or forearm) on a surface such as an armrest of a chair ora table, while using a distal portion as the control entity. Thus, theselection may be performed by the user's hand, fingers or forearm, orcombinations thereof without tiring the arm.

To illustrate additional gestures by way of example and not oflimitation, dropping the control entity downwards could be interpretedas a command to begin a new line, a new paragraph, or to input ahighlighted symbol. Hovering the control unit longer than an activationthreshold may stimulate an autocomplete feature to offer suggestions tocomplete a word or a sentence, thus causing several shortcuts to beshown on the display. In the context of the present application andclaims, the term “shortcut” herein refers to an option on a remoteinformation input interface that appears upon user interaction, whichoffers a choice that is available in the current state, or context, ofthe computer application. That is to say, a user interface element, e.g.a shortcut, may be mapped to an operation to be performed upon selectionof the shortcut. For example, the operation may cause the word shown inthe shortcut to be input to the remote device. Common examples of suchshortcuts include a context menu that appears upon a mouse clickoperation such as a “right-click”, and an iconic link whose activationtriggers some function in a remote device. The activation threshold maybe customized to vary according to specific symbols and applicationscenarios. The display may be altered to reflect potential likelihood orrelevance of a next letter, group of letters, or symbol based upon ananalysis of previously input symbols, e.g., by highlighting likelysymbols on the display. A subset of letters may be shown on the displayto simplify navigation, and may themselves be linked to shortcuts. Therelative location of the subset of letters may be shifted on the displayso as to enable the user to select a preferred letter in a singleselection motion. Typically, the relative location of the subset ofletters is altered without any change to the order of the letters.

In yet other embodiments, the distance and motion relative to thedisplay unit, rather than to the sensing unit, are used for interpretingthe commands.

System Description.

Turning now to the drawings, reference is initially made to FIG. 1,which is a schematic, pictorial illustration of a system 10 forgesture-mediated remote information input, in accordance with anembodiment of the present invention. System 10 incorporates a sensingdevice 12, typically a three-dimensional camera, which detectsinformation that includes the body (or at least parts of the body) of auser 14 or other control entities for controlling a computer applicationas described hereinabove. For the purposes of illustration, the controlentity is described using an example of a hand 16 for providing remoteinput to system 10. In gaming applications, such control entities couldinclude portions of objects being manipulated by user 14, e.g., ashockey sticks, golf clubs, bows, and tennis rackets. The arrangementdescribed in commonly assigned application Ser. No. 12/352,622, filedJan. 13, 2009, which is hereby incorporated by reference, is suitablefor use in system 10 as sensing device 12. Other known three-dimensionalcameras may also be employed as sensing device 12. Although thisembodiment relates to one particular system for providing remoteinformation input, the principles of providing remote information inputthat are implemented in system 10 may similarly be applied, mutatismutandis, in other types of remote information input or gesture controlsystems, using other techniques for providing remote information inputor remote control via specialized gesture.

Information detected by sensing device 12 is processed by a remotedevice 18, which drives a display screen 20 accordingly. Sensing device12 is connected to remote device 18 via a sensing interface 22, whichmay comprise a Bluetooth® adapter, an Infrared Data Association (IrDA)device, a cable connection, a universal serial bus (USB) interface, orany communication interface for outputting sensor data that allowsremote device 18 to import remote sensing data. Remote device 18typically comprises a general-purpose computer processor, which isprogrammed in software to carry out the functions described hereinbelow.The software may be downloaded to the processor in electronic form, overa network, for example, or it may alternatively be provided on tangiblestorage media, such as optical, magnetic, or electronic memory media.Alternatively or additionally, some or all of the image functions may beimplemented in dedicated hardware, such as a custom or semi-customintegrated circuit or a programmable digital signal processor (DSP).Although remote device 18 is shown in FIG. 1, as a separate unit fromsensing device 12, some or all of the processing functions of remotedevice 18 may be performed by suitable dedicated circuitry within thehousing of sensing device 12 or otherwise associated with sensing device12.

Display screen 20 presents user interface elements comprising a pointer24 and a remote information input interface 26, which comprises symbols28, 30, 32, 34, 36. A display interface 38 connects display screen 20 toremote device 18, and may comprise a Bluetooth® adapter, an IrDA device,a cable connection, or any communication interface for outputting imagedata that allows remote device 18 to export visual display data, e.g.,in the form of a compressed image. The symbol selection layout providesa simplified example for the purposes of illustration. In the presentexample, symbols 28, 30, 32 represent numerals, whereas symbols 34 and36 represent actions. Each symbol may be remotely selected or actuatedto control the computer application. Remote information input interface26 may also comprise a zoom level indicator 40 to provide a visualindicator of the zoom level of remote information input interface 26.Zoom level indicator 40 may be shown as a slider, similar to slidersutilized in web browsers and other applications. The zoom level istypically allowed to range within certain limits, e.g., from 50% to500%. In the example of FIG. 1, zoom level indicator 40 shows an initialvalue of 100%. A scale indicator 42 shows symbols 28, 30, 32 having aninitial height of one unit on remote information input interface 26.

Specialized Gestures.

Reference is now additionally made to FIG. 2, which is a view ofportions of system 10 (FIG. 1) operating under remote control of user 14in accordance with an embodiment of the present invention. FIG. 1 andFIG. 2 may be viewed as a sequence of actions. In FIG. 1, user 14 isabout to perform a specialized gesture, and in FIG. 2 completes thegesture. User 14, who is typically viewing display screen 20, performsthe specialized gesture, e.g., a clockwise circular gesture from theperspective of user 14, using hand 16 as described on a referencecoordinate system 44 by directed broken lines 46. Alternatively, othertypes of hand movements may be used to invoke image zoom and otherfunctions, such as moving the hand toward and away from the displayscreen. In reference coordinate system 44 the x, y, and z axes are to beinterpreted as horizontal, vertical, and depth coordinates,respectively, with respect to sensing device 12. Thus, the distancebetween hand 16 and sensing device 12 vary as the specialized gesture isperformed.

The inclination of the plane of the circle may be significant, accordingto specifications programmed in remote device 18. Thus, a circledescribed vertically in the yz plane may be interpreted by remote device18 differently from a circle in the xy plane or a horizontally executedcircular gesture, e.g., as a pointing command. Substantially circulargestures described in various oblique planes may be given even morespecialized interpretations. The clockwise circular gesture isrecognized by sensing device 12, and remote device 18 interprets thegesture as a zoom command. The clockwise circular gesture thus commandsremote device 18 to smoothly increase the zoom (or magnification) levelof remote information input interface 26 on display screen 20 usingpointer 24 as a reference point for the zooming. That is to say, remotedevice 18 identifies an area of display screen 20 around pointer 24 ashaving been selected by user 14 for the zoom command. By performing thegesture, hand 16 may move from an initial position 48 to a finalposition 50. Scale indicator 42 shows symbols 28, 30, 32 having a finalheight of 1.5 units on remote information input interface 26.

In a similar fashion, counter-clockwise circular gestures may beinterpreted by remote device 18 as a command to decrease the zoom level.In the example of FIG. 2, zoom level indicator 40 shows that, incomparison with FIG. 1, the clockwise circular gesture has increased thezoom level from the initial value of 100% immediately prior to thegesture to a final value of 150%. A corresponding increase in the sizeof symbols 28, 30, 32 is shown on display screen 20 as a result of thezoom command. That is to say, one or more of the user interface elementsappearing in the selected area on display screen 20 are magnified.

Reference is now made to FIG. 3, which is a view of portions of system10 (FIG. 1) operating under control of user 14 in accordance with anembodiment of the present invention. In the example of FIG. 3, user 14performs a leftward, substantially horizontal gesture using hand 16 asindicated by an arrow 52 on reference coordinate system 44. The leftwardgesture commands remote device 18 to move pointer 24 to the left onremote information input interface 26. In the example of FIG. 3, theleftward movement of pointer 24 (FIG. 2) executes from a first position(indicated by a cursor 54 (shown in broken outline) to a secondposition, indicated by a cursor 56 (shown in solid outline) as a resultof the command. By performing the gesture, hand 16 may move from aninitial position 58 to a final position 60.

A rightward gesture may be interpreted as a command to move pointer 24to the right from the perspective of user 14, while gestures performedupward and downward may similarly be interpreted by remote device 18 ascommands to move pointer 24 upward and downward, respectively.

Suitable calibration of sensing device 12 and remote device 18 assures adesired sensitivity, i.e., a correspondence between a spatiotemporaldisplacement of the control entity and the effect on elements shown onremote information input interface 26. It is recommended to compensatefor the viewing distance and viewing angle of user 14 using knownmethods. The compensation techniques described in U.S. PatentApplication Publication No. 2009/0009593, entitled “Three-dimensionalProjection Display” may be applied for this purpose.

Remote Information Input.

Reference is now made to FIG. 4, which is a schematic, pictorialillustration of system 10 (FIG. 1) for remote information input, inaccordance with an embodiment of the present invention. A first symbolarc 62 is shown within remote information input interface 26 on displayscreen 20, in an embodiment that implements a T9® text input layout. T9text input represents “text on 9 keys,” a method for streamlining inputof text on numeric keypads, typically for mobile devices, available atthe T9 web site (t9.com). Many suitable variations will occur to thoseskilled in the art for streamlining information input by providing animproved symbol layout on remote information input interface 26. Firstsymbol arc 62 comprises an arcuate, nearly semi-circular display ofgroups of alphanumeric symbol buttons 64 to simulate relaxed movement ofhand 16 while user 14 sits comfortably on a chair 66. First symbol arc62 may comprise additional symbol buttons 68 to support input of specialsymbols, e.g., space, backspace, or carriage return.

As user 14 moves a control entity, such as hand 16 usually, but notnecessarily while seated, and typically in a horizontal arc-likemovement as indicated by an approximately semicircular arc 70 adjacentto reference coordinate system 44. Sensing device 12 detects themovement and remote device 18 interprets the movement by highlightingeach of additional symbol buttons 68 and alphanumeric symbol buttons 64sequentially as hand 16 moves through semi-circular arc 70 from a firstposition 72 to a second position 74. Provision of an arcuate displayenables hand 16 to move while an elbow 118 of the same upper extremityas hand 16 rests on chair 66. A portion of a sphere 122 is shaded withina spherical coordinate system 124 using an axis based upon elbow 118 toindicate an approximate range of motion of hand 16 when elbow 118 restson a surface. In the arrangement of FIG. 4, a three-dimensional space ismapped to spherical coordinate system 124, and is also mapped to atwo-dimensional coordinate system. The latter can be convenientlyappreciated as a plane in reference coordinate system 44.

It is recommended that the movement of semi-circular arc 70 be parallelto the xy plane in reference coordinate system 44. However, the movementmay also be made so that the angle between a plane of the motion ofsemi-circular arc 70 and the xy plane is above 0 degrees, typically upto 45 degrees. In the example of FIG. 4, an emphasized symbol button 76is shown on display screen 20 to indicate that performance of a secondgesture, described hereinbelow as a selection gesture, will result inselection of the symbols displayed therein. That is to say, remotedevice 18 identifies emphasized symbol button 76 as the area of displayscreen 20 that is currently selected by user 14. Corresponding selectiongestures performed by hand 16 at other points along semi-circular arc 70would select corresponding symbols of first symbol arc 62. The firstgesture and the second gesture may be recognized by remote device 18according to time-varying coordinates on the two-dimensional coordinatesystem and the spherical coordinate system, respectively.

Reference is now additionally made to FIG. 5, which is a view ofportions of system 10 (FIG. 1) operating under remote control of user 14in accordance with an embodiment of the present invention. The selectiongesture is typically performed in two stages, as described hereinabove.User 14 may perform a first stage of the selection gesture by movinghand 16 downward in a vertical arc 78 generally directed toward displayscreen 20. Remote device 18 uses the threshold, described hereinabove,to determine completion of each selection stage. The selection gesturemay pivot about the elbow or shoulder, whichever is applicable. Ofcourse, when pivoting about the shoulder the advantages of resting aportion of the arm on a surface are lost. By performing the selectiongesture, hand 16 may move from an initial position 80 to an intermediateposition 130 while performing the first stage, and then to a finalposition 82 while performing a second stage. Sensing device 12 detectsthe movement, and remote device 18 interprets the selection gesture as acommand to display a second symbol arc 84 directly below first symbolarc 62, comprising individually delineated symbol buttons 86, which aregrouped together in highlighted symbol button 76. In the presentexample, user 14 next moves hand 16 in another arc-like movement, whichis detected by sensing device 12. Remote device 18 interprets themovement by highlighting each of individually delineated symbol buttons86 as described hereinabove. User 14 may then perform the second stageof the selection gesture by moving hand 16 further downward to remotelyinput one of individually delineated symbol buttons 86. The second stageof the selection gesture is indicated by a further downwardly directedvertical arc 120. Hand 16 may move from intermediate position 130 tofinal position 82 while performing the second stage. FIG. 4 and FIG. 5may be viewed as a sequence of actions, whereby in FIG. 4 user 14selects an area of display screen 20, e.g., one of additional symbolbuttons 68, and in FIG. 5 selects one of the user interface elements,e.g., highlighted symbol button 76, in order to display second symbolarc 84 and to input one of individually delineated symbol buttons 86.

Use of arcuate displays like first symbol arc 62 for remotely inputtinginformation may provide particularly enhanced ergonomic value. Themotions involved in their use for remote information input are notfatiguing, e.g., in comparison with a standard “QWERTY” keyboard layout.Virtual keyboard layouts such as the QWERTY keyboard layout may not asconveniently permit remote information input with a resting or fixedelbow position.

As noted above, the selection gesture is made by moving hand 16downwards. That is, it involves a forward displacement of the hand inthe z-axis with respect to sensing device 12. There are two variants ofthe motion. In one case user 14 may move hand 16 towards the center(origin) of spherical coordinate system 124 (FIG. 4), such that themotion involves both a component in the xy plane and a component in yzplane. In the other case, user 14 may move hand 16 directly downwards,such that only a component in the yz plane exists. In both cases, remotedevice 18 is able to distinguish the selection gesture from other linearor curved movements in the xy plane alone, by calculating thedisplacement of the hand along the z-axis. When the magnitude of themotion component in the z-axis is greater than another predefinedthreshold, remote device 18 interprets the movement as a selectiongesture and ignores the motion component in the xy plane. In both cases,remote device 18 may provide enhanced ergonomic value when recognizingthese selection gestures, as they allow user 14 to use a naturalselection motion, as indicated by the location of hand 16 on the xyaxis.

If user 14 causes pointer 24 (FIG. 1) to hover over second symbol arc84, remote device 18 may cause a corresponding user interface element,e.g., shortcuts 96, 98 to be shown on remote information input interface26, offering suggestions for completing a word. In the example of FIG.5, letters “S” and “A” have been previously input, and the autocompletefeature of remote device 18 provides shortcuts 96 and 98 for selection.Previously input information may be emphasized on shortcuts 96 and 98.

Reference is now made to FIG. 6, which is a view of portions of system10 (FIG. 1) operating under remote control of user 14 in accordance withan embodiment of the present invention. A domain-specific language model132 may be used to determine the probability of a symbol or next letterbeing preferred by user 14. In the example of FIG. 6, language model 132is shown as a computer program module operated by device 18. In theexample of FIG. 6, user 14 has previously input the letters “INVENTIO”.User 14 next moves hand 16 over emphasized symbol button 76. Device 18uses the domain-specific language model and determines that theprobability of a letter “N” is significantly higher than another lettershown in emphasized symbol button 76, as shown in a shortcut 126. Insome embodiments device 18 uses the domain-specific language model toshift an adjusted second symbol arc 128 to place the letter with ahighest probability beneath emphasized symbol button 76. Both stages ofthe selection gesture may be performed by user 14 in a continuous motionto select a preferred letter, thus minimizing required movement by hand16. In the context of the present application and claims, the term“language model” herein refers to any suitable statistical model forassigning a probability to a sequence of letters or words by means of aprobability distribution.

Reference is now made to FIG. 7, which is a view of portions of system10 (FIG. 1) operating under remote control of user 14 in accordance withan embodiment of the present invention. User 14 may choose to perform adeselection gesture after inputting information remotely, or to deselectsecond symbol arc 84. The deselection gesture may comprise raising hand16 as indicated by an upwardly directed vertical arc 100 which is areversal of vertical arc 78 (FIG. 5). Sensing device 12 detects thedeselection gesture, and remote device 18 interprets the movement as acommand to cancel the selection of second symbol arc 84 (FIG. 5) whichhas responsively been removed from remote information input interface 26in FIG. 7. Thereafter, any movement by hand 16 after the deselectiongesture has been performed is interpreted by remote device 18 as acommand to resume highlighting alphanumeric symbol buttons 64 on firstsymbol arc 62 for selection. In performing the deselection gesture, hand16 moves from an initial position 102 to a final position 104.

In alternative embodiments, after user 14 performs the above-describedselection gesture, remote device 18 automatically removes second symbolarc 84 without requiring the deselection gesture to be performed. Thus,remote information input requires less movement by user 14 than in theprevious embodiment. Device 18 typically requires user 14 to return hand16 to final position 104 before recognizing a new selection.

Embodiments of the present invention that utilize the T9 text inputlayout as symbol arcs on remote information input interface 26 mayprovide an advantage whereby input is provided remotely without the needto move the control entity in three dimensions. As describedhereinabove, moving hand 16 in an arcuate motion along semi-circular arc70 (FIG. 4) is interpreted by remote device 18 as movement within twodimensions, e.g., leftward, rightward, upward and downward. By limitingsemi-circular arc 70 to motions substantially parallel to the xz plane,3-dimensional interpretation issues are avoided. A complex movement inthree dimensions, e.g., to perform the point-and-click gesture, is notrequired, thus simplifying interpretation of the gesture and therebyfacilitating remote information input.

Remote Interfacing via Specialized Gestures.

Reference is now made to FIG. 8, which is a flow chart of a method forremotely interfacing with a computer system, in accordance with anembodiment of the present invention. Shown by way of example, user 14(FIG. 1) may need to search a large volume of media without using aphysical keyboard or other interface connected to a remote device. User14 would thus need to perform efficient, streamlined search commands tointeract remotely with a computer application running on the remotedevice. The process steps are described below in a particular linearsequence for clarity of presentation. However, it will be evident thatsome of them can be performed in parallel, asynchronously, or indifferent orders. The process can be performed, for example, by system10.

User interface elements comprising a remote information input interfaceto a computer application are presented to a user on a display screen ina display presentation step 106. The computer application may be a mediasearch and presentation system. It is assumed that the computerapplication has been loaded, and that a three-dimensional sensing deviceis in operation. The sensing device can be any three-dimensional sensoror camera, provided that it generates data for interpretation by theremote device.

The user performs a first gesture in a three-dimensional space using acontrol entity, e.g., a part of the user's body. A sensing device, suchas sensing device 12 (FIG. 1), detects the gesture made by the controlentity, e.g., hand 16, in a gesture detecting step 108. The computeriteratively analyzes three-dimensional data provided by the sensingdevice, for example by constructing a three-dimensional map as describedin commonly assigned co-pending U.S. application Ser. No. 12/683,452,which is herein incorporated by reference. In response to the detectedgesture, an area of the display screen is identified by the computer ina selected area identification step 110.

The first gesture is recognized by the computer as a command to increasethe magnification level of user interface elements within the selectedarea on the display screen in a magnification level adjusting step 112.Any gesture recognition algorithm may be employed to carry outmagnification level adjusting step 112, so long as the system can relatethe user gesture to a recognized command and a location of interest onthe remote information input interface.

A second gesture is recognized by the computer as a command to selectone of the user interface elements within the selected area in aselection gesture detecting step 114. The second gesture can be for anypurpose, for example to perform another zoom command, to input a symbol,or to alter the remote information input interface in accordance withthe gesture identified. For example, the clockwise circular gesturecommand described with respect to FIG. 2 might correspond to aninstruction to increase the zoom level of the remote information inputinterface on the display screen, while a counter-clockwise circulargesture, in which the direction of the motion is reversed, could resultin an instruction to decrease the zoom level. Many such combinationswill occur to a developer of computer applications or other signalprocessing systems. An updated display screen results, and is shown insubsequent iterations of the method. In practice the process iterates solong as the remote device is active or some error occurs.

The method then terminates at a final step 116.

Alternative Embodiments

In some embodiments, the circular gestures comprise requiring at leastone complete circle to be performed by the control entity before thezoom level is changed. In alternative embodiments, multiple controlentities are used to perform the specialized gesture. For example, thezoom command may be input using a second hand (not shown) to complementhand 16 (FIG. 1). Once remote device 18 recognizes the second hand byanalyzing input from sensing device 12, movement of the second handfarther away from hand 16 may be interpreted as the zoom command toincrease the zoom level, and vice versa. Using multiple control entitiesmay provide an advantage wherein pointer 24 is not moved prior to thechange in zoom level. Thus, the changes in magnification may beperformed around pointer 24.

In variants of the embodiments of FIG. 6 and FIG. 7, language model 132is used to assign a probability to each letter on remote informationinput interface 26. Device 18 may order the letters accordingly,displaying the letters in a continuous ungrouped series of individualletters, rather than in groups, e.g., the group of three lettersdisplayed in emphasized symbol button 76. Device 18 may invite access toletters on the display having relatively high probabilities, e.g., bypresenting them in closer proximity to the center of remote informationinput interface 26. Alternatively, letters having relatively high andlow probabilities may be grouped together and placed into secondarysymbol arcs (not shown).

In yet another variant, the spatial distribution of letters in a symbolarc may reflect their respective probabilities. Thus, letters havingrelatively high and low probabilities of selection may be spaced apartand crowded together, respectively, in varying degrees.

Other commercial methods for remote information input may be used inconjunction with the specialized gestures and command interpretation byremote devices using three-dimensional sensing described hereinabove.For example, concepts described by the MessagEase™ text input system,available for sale at the MessagEase web site (exideas.com), may beenhanced accordingly.

In an alternative embodiment (not shown specifically in the figures),text input created by the user is shown as a linear stream of charactersrunning across the screen, from left to right, for example. The user mayperform a special gesture, recognized by the remote device, to mark aneutral reference position. Movement of the user's hand in a directionalong the sequence, such as to the right of the reference position willthen cause the display to advance to the right along the text streamwhereas movement to the left will scroll backward through the textstream. The scroll speed presented by the remote device on the displaymay initially be slow when the user gestures sideways to the right orleft and may gradually accelerate the longer the user's hand is in theadvance or reverse position. As the text stream advances, theabove-mentioned language model may be used to display alternativechoices of additional characters and even words to append to the stream.These choices may be displayed above and/or below the existing line ofcharacters, with the likeliest choices typically vertically closest tothe line and possibly magnified. The user selects the desired choice byupward and downward motions of the hand, perpendicular to the directionof the text sequence. Thus, following the initial special gesture, theuser can add text quickly and efficiently using simple right/left andup/down motions.

The user's right/left and up/down hand motions may be made in agenerally planar space or, if the elbow is resting on a surface as insome of the embodiments described above, may be over a generallyspherical surface. In either case, the right/left and up/down motionsare not limited to a two-dimensional plane, but may be mapped to atwo-dimensional coordinate system by the remote device. Within thislatter coordinate system, one dimension of hand movement controls thespeed of scrolling forward and back (wherein backward movement maydelete characters previously appended to the stream), while the otherdimension controls the selection of new characters.

This sort of embodiment may be used to present and add text input in asort of continuous “flight mode”: As the user scrolls to the right(forward) to add text to the stream being created, various potentialcontinuations of the existing text are presented to the right of theexisting text, above and/or below the text line. The potentialcontinuations may be ordered or otherwise presented in such a way thatthe effort necessary to select a given continuation is a decreasingfunction of the likelihood that the given continuation is the correctone, based on the language model (higher likelihood yields lowereffort). For example, likelier continuations may be presented withlarger size and/or in closer proximity to the current cursor position.

It will be appreciated by persons skilled in the art that the presentinvention is not limited to what has been particularly shown anddescribed hereinabove. Rather, the scope of the present inventionincludes both combinations and sub-combinations of the various featuresdescribed hereinabove, as well as variations and modifications thereofthat are not in the prior art, which would occur to persons skilled inthe art upon reading the foregoing description.

1. A method for operating a computerized system, comprising: presentinguser interface elements on a display screen of the computerized system;detecting respective movements of two hands of a user in athree-dimensional space, including a gesture in which one of the handsmoves away from the other of the hands in the three-dimensional space;and responsively to the gesture, increasing a magnification level of oneor more of the user interface elements appearing in an area on thedisplay screen.
 2. The method according to claim 1, and comprisingdetecting, in the respective movements prior to the gesture, an initialgesture comprising the other of the hands moving from a first positionto a second position in the three-dimensional space, and identifying thearea on the display screen responsively to the initial gesture.
 3. Themethod according to claim 2, and comprising presenting a pointer on thedisplay screen, and moving the pointer responsively to the initialgesture, and wherein the identified area comprises an area around thepointer.
 4. The method according to claim 2, and comprising highlightinga given user interface element responsively to the initial gesture, andwherein the one or more of the user interface elements comprises thehighlighted user interface element.
 5. The method according to claim 1,and comprising detecting, in the respective movements subsequent to thegesture, an additional gesture made by the other of the hands, andselecting one of the one or more of the user interface elementsresponsively to the additional gesture.
 6. The method according to claim5, wherein increasing the magnification level comprises at least one ofexposing new details on the display and exposing relevant options on thedisplay.
 7. The method according to claim 6, wherein selecting the oneof the one or more user interface elements comprises selecting a givenrelevant option.
 8. The method according to claim 1, and comprisingdetecting, in the respective movements subsequent to the gesture, anadditional gesture comprising moving the one of the hands toward theother of the hands in the three-dimensional space, and decreasing themagnification level of the one or more of the user interface elementsresponsively to the additional gesture.
 9. The method according to claim1, wherein detecting the respective movements comprise detectingthree-dimensional locations of the hands and mapping, by thecomputerized device, the three-dimensional locations from thethree-dimensional space to a two-dimensional coordinate system.
 10. Anapparatus, comprising: a display screen; a sensing device configured todetect respective movements of the two hands of a user in athree-dimensional space; and a processor configured: to present userinterface elements on the display screen; to recognize, in therespective movements, a gesture in which one of the hands moves awayfrom the other of the hands in the three-dimensional space; andresponsively to the gesture, to increase a magnification level of one ormore of the user interface elements appearing in an area on the displayscreen.
 11. The apparatus according to claim 10, wherein the processoris configured to detect, in the respective movements prior to thegesture, an initial gesture comprising the other of the hands movingfrom a first position to a second position in the three-dimensionalspace, and to identify the area on the display screen responsively tothe initial gesture.
 12. The apparatus according to claim 11, whereinthe processor is configured to present a pointer on the display screen,and to move the pointer responsively to the initial gesture, and whereinthe identified area comprises an area around the pointer.
 13. Theapparatus according to claim 11, wherein the processor is configured tohighlight a given user interface element responsively to the initialgesture, and wherein the one or more of the user interface elementscomprises the highlighted user interface element.
 14. The apparatusaccording to claim 10, wherein the processor is configured to detect, inthe respective movements subsequent to the gesture, an additionalgesture made by the other of the hands, and to select one of the one ormore of the user interface elements responsively to the additionalgesture.
 15. The apparatus according to claim 14, wherein increasing themagnification level comprises at least one of exposing new details onthe display and exposing relevant options on the display.
 16. Theapparatus according to claim 15, wherein the processor is configured toselect the one of the one or more user interface elements by selecting agiven relevant option.
 17. The apparatus according to claim 10, whereinthe processor is configured to detect, in the respective movementssubsequent to the gesture, an additional gesture comprising moving theone of the hands toward the other of the hands in the three-dimensionalspace, and to decrease the magnification level of the one or more of theuser interface elements responsively to the additional gesture.
 18. Theapparatus according to claim 10, wherein the processor is configured todetect three-dimensional locations of the hands and to map thethree-dimensional locations from the three-dimensional space to atwo-dimensional coordinate system.
 19. A computer software productcomprising a non-transitory computer-readable medium, in which programinstructions are stored, which instructions, when read by a computer,cause the computer: to present user interface elements on a displayscreen of the computerized system, to to detect respective movements oftwo hands of a user in a three-dimensional space, including a gesture inwhich one of the hands moves away from the other of the hands in thethree-dimensional space; and responsively to the gesture, to increase amagnification level of one or more of the user interface elementsappearing in an area on the display screen.
 20. The computer softwareproduct according to claim 19, and comprising additional programinstructions, which when read by the computer, cause the computer todetect, in the respective movements subsequent to the gesture, anadditional gesture comprising moving the one of the hands toward theother of the hands in the three-dimensional space, and to decrease themagnification level of the one or more of the user interface elementsresponsively to the additional gesture.